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THE DIMENSIONS OF PROGRAM EVALUATION ARE OUTLINED, SOME 
NON-TEST METHODS WHICH CAN BE USED ARE DESCRIBED, AND THE 
IMPORTANCE OF MONITORING AND CONTROLLING THE INTERVENTION ARE 
STRESSED. THE IMPORTANCE OF TEAMWORK BETWEEN EVALUATORS AND 
INTERVENTIONISTS IS ILLUSTRATED AND THE VALUE OF SUCH AN 
APPROACH IN FURTHERING THE CLEAR CONCEPTUALIZATION AND 
REFINEMENT OF PROGRAMS IS DISCUSSED. SOME PROBLEMS RELEVANT 
TO PROGRAMS FOR DISADVANTAGED CHILDREN ARE EXAMINED. THE 
POTENTIAL CAPACITIES OF DISADVANTAGED CHILDREN ARE NOT 
REFLECTED BY PAST AND PRESENT PERFORMANCE. IN EVALUATING THE 
EFFECTIVENESS OF INTERVENTION, THE REACTIONS OF THE 
ENVIRONMENT AS WELL AS THE RESPONSES OF THE SUBJECT MUST BE 
EXAMINED. UNLESS FAVORABLE CHANGES IN THE INDIVIDUAL ARE 
SUPPORTED BY THE ENVIRONMENT, SUCH CHANGES ARE UNLIKELY TO BE 
MAINTAINED. (WR) 
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Ubenever ve ijndertake any kind of intervention whether it is teaching, 
training, counselling, therapy, consultation or the institution of seme new 
administrative procedure, we should he concerned with how it works, T?e 
evaluate our activities so that we can discard those innovations which do 
not live up to ou* -ctation, make sure that programs are going accord- 
ing to plan and ref. iniprove those methods which show premise, "!Jho, 

why, what, where, and how? ' are a,s applicable to evaluation as to jour- 
nalism, Ue must know what the conditions are that we plan to change, who 
are involved, what we plan to do, whether the plans are carried out and 
what their effect is. Thus, there are three stages in evaluation; the 
original conditions, the process of intervention and the outcome, (l) 

In this paper we will briefly review some aspects of all three stages and 
consider in greater detail the evaluation of the intervention process. 
Stages in Evaluation 

The Baseline , In order to establish the baseline for any program 
and its evaluation, it is necessary to know the basic characteristics of 
the population and their present functioning; the characteristics of the 
interventionists; the setting of the program and the identification of the 
need for change. 
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Intervention Process > In order to understand the process of inter- 
vention, we mu£ "know how change takes place; what is undertaken; how it 
is carried cut; wnat the inanediate responses and interactions are; 
whether the program is going according to plan; if ehurxges are made; 
why they are made and what unanticipated side effects mary have occurred. 

The Outcome > In evaluating outcome, we must look at changes in the 
population and the setting. Are they according to plan; if so, do these 
changes produce the results we anticipated; if not, hew not and why not. 

The Research De sign 

In establishing our research design, certain facts have to be kept in 
mind. Resources are always in short supply, A total evaluation would 
probably cost about 20 times as much as the program itself. Consequently, 
research should be looked upon as an evolving process, We must establish 
the questions which have priority for us now and recognize that one set 
of questions lead to another and that sometimes there are questions we 
cannot tackle until we have obtained some prior answers or have tried 
out new methods. 

Secondly, the mere fact of evaluating a program may affect it. It 
has long .on recognized that merely paying attention to a person or a 
program alters the situation. Introducing new, different, or additional 
people affect the climate of a program. We must be careful to keep an 
eye on the effects of evaluation on the program and not to swamp it with 
tests, questionnaires, checklists, or observers. 
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Teamwork Between E valuators and Interventionists 

As far as possible, the interventions which are planned and the 
chsoiges which ere expected should be derived from theoretical premises 
and should be defined in operational, behavioral terms. For instance, 
if we say that our goal is to help someone become less tense or enable 
them to perform better, this is a subjective judgement which cannot be 
impartially assessed. However, if we say that we wish to enable a child 
to be able to remain absorbed in one task for ten, fifteen, or twenty 
minutes, rather than moving restlessly from task to task every one or 
two minutes, we can easily judge whether we have been successful. 

The program people who are undertaking the intervention should be 
responsible for the conceptualizations of their program and the clear 
definition of their methods or the changes they hope to achieve. They 
must again be able to define their interventions in behavioral terms. 

They will usually also choose their own population ancl se*;ting and who 
will undtirtake the intervention. Beyond this, however, the establish- 
ment of a baseline and the assessment of outcome may be undertaken by an 
entirely r>eparate teem, even thou^ the program people may be engaged in 
collectirg certain data for the evaluators. 

IJhen it comes to the evaluation of the Intervention Process, however, 
we find it much more dynamic and profitable to consider interventionists 
and evaluators as partners on the same team. 

The Intervention Process 

The purposes of studying the process of intervention are three-fold: 
firstly, to know what goes on in the interaction between the interventionist 
and the subjects; secondly, to assist in ensuring that the program is 
carried out according to plan; and thirdly, when responses are fouiad to be 
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different from those expected, to assist in refining and improving the 
program. Thus such evaluation becomes the ongoing tool of the interventionists. 

Both the evaluators and the interventionists will probably participate 
in data collection. The interventi oni et. will state clearly what he jlans 
uO \j,o oinj. what he e^^eCto tvj xiuppen. iie wixl Keep descriptive reco2?d8, 
make ratings, fill out checklists while the evaluators will act as ob- 
servers, analyze tapes, conduct tests which will add to the interventionists * 
data and check on its reliability. 

In order for such a team to function effectively, it is important 
that the evaluators should be looked upon as collaborators and helpers 
and not as spies, finks, or judges. This means that they must meet to- 
gether regularly, that the evaluators must be willing to share their 
results and be able to feedback information speedily enough for it to be 
useful to the interventionist and that the interventionist must strive to 
be undefensive, to look upon their work as an adventure or an e 3 q)loration 
in which knowledge about the problems they encounter will enable them to 
achieve their goals more effectively. Mutual trust has to be developed 
between the two sides of the team and the eveiluators must recognize that 
to remain impartial in such a role is hard and that at times they may be 
no more reliable than the interventionists. Particularly, they have to 
watch their overidentification either with the program or with the subjects 
of the program. Research observers have been known to turn themselves 
into supervisors or ardent protectors of the pupil or patient, ( 2 ) 

V/hen we study the process of intervention and attempt to monitor and 
control program, we have to make choices about the important variables 
to be studied. Shall we concentrate on the interventionist, what kind 
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of impression he makes, what kind of person he seems to he, what he 
does t'nd how he does it? Shall we stiidy the individual subject, his 
roles, actions, attitudes? Or what goes on in the class or group, the 
climate, the interactions, the activities, the changing structure? Or 
possibly we are interested in the overall setting, do changes occur in 
the institution, its progress itc network of ccamuunication, ios ergeni na- 
tion of roles and functions in response to our interventions? Obviously, 
we cannot tackle all of these at once and so we must make decisions 
which are congruent with our theoretical approaches and our goals. 

If our hypothesis is that when a teacher undertakes certain opera- 
tions, she will elicit certain responses from the children, we must study 
both whether she is behaving in the way which was planned and whether the 
children are responding as anticipated; 

In one program we hypothesized that there was a relationship between 
the underachievement of highly gifted youth and the manner in which the 
family interacted and that if we could help the members together deal 
differently with problems, we would find a difference in the youth's 
school performance. (3) We further hypothesized that we could achieve 
this change in family functioning throu^ bringing families together in 
a group for a limited number of sessions in which the group leader would 
focus the discussion on the way the families dealt with their problems. 
We, thus, had to know the school performance of the youth before and 
after treatment and how this congoared with untreated youth. \le needed 
to know how the family dealt with problems before and after treatment and 
we required to ensure that the leader focused the group discussion in the 
desired way. We, therefoirt.*, collected data related to the youth's scores 
on the Lorge-Throndike to indicate that he was giffced. The school record 
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showed his performance. We devised a problem-solving task (the Family 
Classification Test) ( 4 ) which we administered to the family before and 
after treatment and as part of a follow-up study^ and we devised a scale 
for rating the group leader* s interventions in terms of their relevance 
and focus to the task which was used by observers with considerable 
reliability. (3) 

In a second program, we were interested in knowing how mucn was 
learnt in a course and whether this learning was related to the frequency 
of e3q>osure, (5) We found that while it was very hard to tell how much 
had been learnt because information reproduced in a post-test differed 
considerably from the information which the students showed they possessed 
in a pre-test, the information which was reproduced correlated positively 
with the frequency with which such information was transmitted in the 
course. We were able to reach this conclusion through taping the course 
and making a content analysis of the discussion. 

In a job training program for socially disadvantaged youth, we hy- 
pothesized that lack of capacity to take self and group responsibility 
was related to irresponsibility and poor performance on the job. We 
used incidence of absenteeism and lateness and teacher ratings of at- 
tentiveness on the job as indices of job responsibility and decided that 
practice in decision-making around job-related problems would be an 
appropriate training measure. (6) \Je thus had to collect on the one 
hand the stavtistlcs on job attendance and the teacher ratings on the 
other we had to monitor the program to ensure that the students were dis- 
cussing job-related issues, were taking part in the discussions and that 
the group members were showing an increased capacity to reach consensus. 

A group observer, therefore, collected the number of Issues discussed, 
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abstracted the interactions from tapes, had these rated for degree of con- 
sensus by independent raters and charted the spread of participation amongst 
group members. 

In developing practicum for new employment roles, we are interested 
in whether supervisors are giving the trainees the opportxmity to perform 
and learn the tasks which are outlined on a job description. Consequently, 
we ask our supervisors and trainees to fill out checklists of the tasks 
which have been performed and our evaluators take sample observations of 
trainees on the job to check for the reliability of the lists. If we find 
that trainees are not being afforded adequate experience, we discuss 
the problem in joint meetings, which may result in the supervisors re- 
organizing their practices or in a change in the job description. In a 
classroom aide training program such quality control has resulted in con- 
tinued refinement of the job description of the aide. 

We were interested in knowing in a counsellor training program 
whether we were teaching the counsellors what we thought we were teaching 
and in the designated manner and in whether the counsellors were conduct- 
ing their counselling groups along the lines which we had planned. We 
consequently studied the degree and quality of participation and the 
content in both teaching and counselling groups by means of content 
analysis of tapes, ratings of sample interactions and spot observations. (7) 

In an institutional setting where we are interested in the authority 
structure, we are studying the way in which program is planned, who is 
involved, how plans are discussed, how decisions are made and who makes 
them, through an interaction analysis of planning meetings. 
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Methods of Data Collection 

Baseline data is obviously crucial to the evaluation of change, for 
if we do not know what we started with, we cannot hope to evaluate the 
effectiveness of our interventionc We must axso know or in some way be 
able to estimate what woiad happen if we did not intervene or if we con- 

Llxiued to iu uOiila^xy V/ayS# VVti mUS'O 0.6C1Q.6 Wlia*U T/iie rtiitivfcixii/ 

variables are and collect our data which may be concerned with standard 
sociological facts, present performance which can be observed or tested 
and attitudes and feelings which may be checked, obsei^ed or probed for. 
Thus, we may use questionnaires, structured or semi-structured interviews, 
checklists, ratings or assessments by self or others, life historicis, 
or assessments of the way :ln which simulated or real tasks are performed. 

In one program where we were interested in estimating the attitudes 
of trainees to disadvantaged populations and their sensitivity to group 
operations at the start of a program, we asked them to fill out question- 
naires, to take part in a group discussion related to problems of disad- 
vanted youth, and to fill out a questionnaire which described the group 
operations . 

The examination of the interaction between interventionist and subject 
or group of subjects is particularly crucial and also particularly diffi- 
cult for it is not the content but also the quality of the transaction 
which is important and a great deal of communication is transmitted non- 
verbally by gestures and experience, rather than in words. Althougn a 
number of researchers have been studying this process, our instruments 
are still extremely primitive. Again, choices have to be made. Bales, 
for instance, has defined and refined a method for analyzing each group 
intervention in terms of 12 categories. (9) In one of our studies, we 
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have concentrated on the actions of the interventionist. ( 3 ) Berne has 
been interested in the identification of interchange sequences (transac- 
tions) which he terms "games." (lO) Stock and Lieberman have tried to 
isolate critical issues" in therapy groups which can be examined over- 
time. Leavy has tried to rate the feeling-tone of the participants in (11 ) 
terms such as dominating, bossy, compliant. (12) 

In the evaluation of intervention process, in our different programs, 
we have used sociometric data, diary records, checklists of activities, 
ratings of behavior, interaction scales concerned with qutmtity and type 
of interaction, content analysis of interaction and of critical incidents, 
issues or decisions, and expressions of attitudes by participants. 

In some situations we have used depth studies of typical cases or si- 
tuations in order to ^j.n greater understanding or for the development of 
hypotheses . 

We have gauged our outcomes through studies of behavior as it related 
to our predictions. We have designed simulated scenes and tasks, obtained 
evaluations of teachers, supervisors, administrators and students. Used 
semi-objective criteria such as grades, promotions, capacity to hold the 
job and other achievements. 

We should now like to discuss very briefly some of the problems which 
are paiijicularly relevant to programs for disadvantaged children 

First, we have to recognize that past and present performance do not 
reflect the potential capacities of the children. Achievement and in- 
telligence tests give information about what the child is like now, but 
little help in predicting what he might be able to achieve. We have also 
come to realize much more vividly that a person* s behavior is conditioned 
not only by his personal characteristics, but also by the circumstances 
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to which he is eoqposed. Thus, a child who may behave and leam well under 
a competent teacher in a calm and structured classroom may fall to pieces 
in another class where he becomes for some unpredi eatable reason a source 
of irritation to the teacher or where she allows a confused and chaotic 
climate to prevail. Schools may have very different capacities to tolerate 
noise, stu'^snt disruption or a sloii pace of learning; and we all know, 
that "drop-outs" may ^uite frequently be termed "push-outs." Consequently, 
past behavior cannot really be used as an accurate assessment of a child’s 
present status unless we also know the degree of stress under which the 
behavior occurred. \le thus find very wide variation of responsiveness 
in children whose original profiles looked rather similar and must con- 
stantly be testing our assumptions and predictions. 

Secondly, we have also learnt that even if we obtain favorable changes 
in the individual, if the environment does not support these new ways or 
worse still, reinforces the old behaviors, such changes are unlikely to be 
maintained. For instance, if we arouse the children’s desire to leam and 
they are still subjected to conditions under which they are bound to fail, 
their eagerness will soon evaporate. More difficult is the problem of 
helping children who have been idle and delinquent in a delinquent world 
to adopt and maintain more socially acceptable and productive habits, for 
this not only means a change in behavior and self-concept, it also means 
a moving away from friends and even family. Similarly, we have found that 
we may train personnel to perform conpetently; but if they are placed in an 
institution which does not accept the same standards and where staff and 
supervisors are accustomed to different methods, our trainees may well be 
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forced to abandon their standards or to leave the system* It is, therefore, 
extremely important in .ar evaluation of program to keep track not only 
of what is fed into the program and the responses of the subjects, but also 
to be concerned with the reactions of the environment within which the 
individual has to perform. 

Summary 

In this paper we have outlined the dimensions of program evaluation, 
described some of the non-test methods which can be used, and stressed the 
importance of monitoring and controlling the intei*\ention itself and in 
making careful choices so that we avoid overload. Me have tried to illus- 
trate how important it is for evaluators and interventionists to work as 
a team and to emphasize the value of such an approach in furthering the 
clear conceptualization and refinement of the program. We cannot over- 
state that without such conceptualization, no conclusions about the ef- 
fectiveness of a program can be drawn and the program itself is not 
repeatable , 

We have also noted that past and present records and achievements are 
not reliable predictors of the potential of socially disadvantaged children 
and that in evaluating the effectiveness of interventions, it is important 
to examine not only the responses of the subjects, but the reactions of 
the environment in which they have to perfoim. In Lippitt*s teims, we 
must be concerned not only with the individual personality system, but 
with the institutional and comraunity systems as well. 
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